Panoramic camera, which captures the 360° view of scenic places such as tourist resorts, is an example of a panoramic imaging system. Panoramic imaging system is an imaging system that captures the views one could get by making one complete turn-around from a given spot. On the other hand, omnidirectional imaging system captures the view of every possible direction from a given position. Omnidirectional imaging system provides a view that a person could observe from a given position by turning around as well as looking up and down. In a mathematical terminology, the solid angle of the region that can be captured by the imaging system is 4π steradian.
There have been a lot of studies and developments of panoramic imaging systems not only in the traditional areas such as photographing buildings, nature scenes, and heavenly bodies, but also in security/surveillance systems using CCD (charge-coupled device) or CMOS (complementary metal-oxide-semiconductor) cameras, virtual touring of real estates, hotels and tourist resort, and navigational aids for mobile robots and unmanned aerial vehicles (UAV).
One method of obtaining a panoramic image is to employ a fisheye lens with a wide field of view (FOV). For example, the entire sky and the horizon can be captured in a single image by pointing a camera equipped with a fisheye lens with 180° FOV toward the zenith (i.e., the optical axis of the camera is aligned perpendicular to the ground plane). On this reason, fisheye lenses have been often referred to as “all-sky lenses”. Particularly, a high-end fisheye lens by Nikon, namely, 6 mm f/5.6 Fisheye-Nikkor, has a FOV of 220°. Therefore, a camera equipped with this lens can capture a portion of the backside of the camera as well as the front side of the camera. Then, panoramic image can be obtained from thus obtained fisheye image after proper image processing.
In many cases, imaging system is installed on vertical walls. Imaging systems installed on outside walls of a building for the purpose of monitoring the surroundings, or a rear view camera for monitoring the backside of a passenger car are such examples. In such cases, it is inefficient if the horizontal field of view is significantly larger than 180°. This is because a wall, which is not needed to be monitored, takes up a large space in the monitor screen. Pixels are wasted in this case, and screen appears dull. Therefore, a horizontal FOV around 180° is more appropriate for such cases. Nevertheless, a fisheye lens with 180° FOV is not desirable for such application. This is because the barrel distortion, which accompanies a fisheye lens, evokes psychological discomfort and abhorred by the consumer.
An example of an imaging system, which can be installed on an interior wall for the purpose of monitoring the entire room, is given by a pan•tilt•zoom camera. Such a camera is comprised of a video camera, which is equipped with an optical zoom lens, mounted on a pan•tilt stage. Pan is an operation of rotating in the horizontal direction for a given angle, and tilt is an operation of rotating in the vertical direction for a given angle. In other words, if we assume that the camera is at the center of a celestial sphere, then pan is an operation of changing the longitude, and tilt is an operation of changing the latitude. Therefore, the theoretical range of pan operation is 360°, and the theoretical range of tilt operation is 180°. The shortcomings of a pan•tilt•zoom camera include high price, large size and heavy weight. Optical zoom lens is large, heavy and expensive due to the difficulty in design and the complicated structure. Also, a pan•tilt stage is an expensive device no cheaper than a camera. Therefore, it cost a considerable sum of money to install a pan•tilt•zoom camera. Furthermore, since a pan•tilt•zoom camera is large and heavy, this point can become a serious impediment to certain applications. Examples of such cases include airplanes where the weight of the payload is of critical importance, or a strict size limitation exists in order to install a camera in a confined space. Furthermore, pan•tilt•zoom operation takes a time because it is a mechanical operation. Therefore, depending on the particular application at hand, such a mechanical operation may not be fast enough.
References 1 and 2 provide fundamental technologies of extracting an image having a particular viewpoint or projection scheme from an image having other than the desirable viewpoint or projection scheme. Specifically, reference 2 provides an example of a cubic panorama. In short, a cubic panorama is a special technique of illustration wherein the observer is assumed to be located at the very center of an imaginary cubic room made of glass, and the outside view from the center of the glass room is directly transcribed on the region of the glass wall where the ray vector from the object to the observer meets the glass wall. An example of a more advanced technology is provided in the above reference with which reflections from an arbitrarily shaped mirrored surface can be calculated. Specifically, the author of reference 2 created an imaginary lizard having a highly reflective mirror-like skin as if made of a metal surface, then set-up an observer's viewpoint separated from the lizard, and calculated the view of the imaginary environment reflected on the lizard skin from the viewpoint of the imaginary observer. However, the environment was not a real environment captured by an optical lens, but a computer-created imaginary environment captured with an imaginary distortion-free pinhole camera.
On the other hand, an imaging system is described in reference 3 that is able to perform pan•tilt•zoom operations without a physically moving part. The said invention uses a camera equipped with a fisheye lens with more than 180° FOV in order to take a picture of the environment. Then, user designates a principal direction of vision using various devices such as a joystick, upon which, the computer extracts a rectilinear image from the fisheye image that could be obtained by heading a distortion-free camera to that particular direction. The main difference between this invention and the prior arts is that this invention creates a rectilinear image corresponding to the particular direction the user has designated using devices such as a joystick or a computer mouse. Such a technology is a core technology in the field of virtual reality, or when it is desirable to replace mechanical pan•tilt•zoom camera, and the keyword is “interactive picture”. In this technology, there are no physically moving parts in the camera. As a consequence, the system response is fast, and there is less chance of mechanical failure.
Ordinarily, when an imaging system such as a security camera is installed, a cautionary measure is taken so that a vertical line perpendicular to the horizontal plane also appears vertical in the acquired image. In such a case, vertical lines still appear vertical even as mechanical pan•tilt•zoom operation is performed. On the other hand, in the said invention, vertical lines generally do not appear as vertical lines after a software pan•tilt•zoom operation has been performed. To remedy such an unnatural result, a rotate operation is additionally performed, which is not found in a mechanical pan•tilt•zoom camera. Furthermore, the said invention does not provide the exact amount of rotate angle, which is needed in order to display vertical lines as vertical lines. Therefore, the rotation angle must be found in a trial-and-error method in order to display vertical lines as vertical lines.
Furthermore, the said invention assumes that the projection scheme of the fisheye lens is an ideal equidistance projection scheme. But, the real projection scheme of a fisheye lens generally shows a considerable deviation from an ideal equidistance projection scheme. Since the said invention does not take into account the distortion characteristics of the real lens, images obtained after image processing still shows distortion.
The invention described in reference 4 remedies the shortcoming of the invention described in reference 3, namely the inability of taking into account the real projection scheme of the fisheye lens used in image processing. Nevertheless, the defect of not showing vertical lines as vertical lines in the monitor screen has not been resolved.
From another point of view, all animals and plants including human are bound on the surface of the earth due to the gravitational pull, and most of the events, which need attention or cautionary measure, take place near the horizon. Therefore, even though it is necessary to monitor every 360° direction on the horizon, it is not as important to monitor high along the vertical direction, for example, as high as to the zenith or deep down to the nadir. Distortion is unavoidable if we want to describe the scene of every 360° direction on a two-dimensional plane. Similar difficulty exists in the cartography where geography on earth, which is a structure on the surface of a sphere, needs to be mapped on a planar two-dimensional atlas. Among all the distortions, the distortion that appears most unnatural to the people is the distortion where vertical lines appear as curved lines. Therefore, even if other kinds of distortions are present, it is important to make sure that such a distortion is absent.
Described in reference 5 are the well-known map projection schemes among the diverse map projection schemes such as equi-rectangular projection, Mercator projection and cylindrical projection schemes, and reference 6 provides a brief history of diverse map projection schemes. Among these, the equi-rectangular projection scheme is the projection scheme most familiar to us when we describe the geography on the earth, or when we draw the celestial sphere in order to make a map of the constellation.
Referring to FIG. 1, if we assume the surface of the earth or the celestial sphere is a spherical surface with a radius S, then an arbitrary point Q on the earth's surface has a longitude ψ and a latitude δ. On the other hand, FIG. 2 is a schematic diagram of a planar map drawn according to the equi-rectangular projection scheme. A point Q on the earth's surface having a longitude ψ and a latitude δ has a corresponding point P″ on the planar map (234) drawn according to the equi-rectangular projection scheme. The rectangular coordinate of this corresponding point is given as (x″, y″). Furthermore, the reference point on the equator having a longitude Wand a latitude 0° has a corresponding point O″ on the planar map, and this corresponding point O″ is the origin of the rectangular coordinate system. Here, according to the equi-rectangular projection scheme, the same interval in the longitude (i.e., the same angular distance along the equator) corresponds to the same lateral interval on the planar map. In other words, the lateral coordinate x″ on the planar map (234) is proportional to the longitude.
Math Figure 1x″=cψ  [Math.1]
Here, c is proportionality constant. Also, the longitudinal coordinate y″ is proportional to the latitude, and has the same proportionality constant as the lateral coordinate.
Math Figure 2y″=cδ  [Math.2]
The span of the longitude is 360° ranging from −180° to +180°, and the span of the latitude is 180° ranging from −90° to +90°. Therefore, a map drawn according to the equi-rectangular projection scheme must have a width W:height ratio of 360:180=2:1.
Furthermore, if the proportionality constant c is given as the radius S of the earth, then the width of the said planar map is given as the perimeter of the earth measured along the equator as given in Eq. 3.
Math Figure 3W=2πS  [Math.3]
Such an equi-rectangular projection scheme appears as a natural projection scheme considering the fact that the earth's surface is close to the surface of a sphere. Nevertheless, it is disadvantageous in that the size of a geographical area is greatly distorted. For example, two very close points near the North Pole can appear as if they are on the opposite sides of the earth in a map drawn according to the equi-rectangular projection scheme.
n the other hand, in a map drawn according to the Mercator projection scheme, the longitudinal coordinate is given as a complex function given in Eq. 4.
Math Figure 4
                              y          ″                =                  c          ⁢                                          ⁢          ln          ⁢                      {                          tan              ⁡                              (                                                      π                    4                                    +                                      δ                    2                                                  )                                      }                                              [                  Math          .                                          ⁢          4                ]            
On the other hand, FIG. 3 is a conceptual drawing of a cylindrical projection scheme or a panoramic perspective. In a cylindrical projection scheme, an imaginary observer is located at the center N of a celestial sphere (331) with a radius S, and it is desired to make a map of the celestial sphere centered on the observer, the map covering most of the region excluding the zenith and the nadir. In other words, the span of the longitude must be 360° ranging from −180° to +180°, but the range of the latitude can be narrower including the equator within its span. Specifically, the span of the latitude can be assumed as ranging from −Δ to +Δ, and here, Δ must be smaller than 90°.
In this projection scheme, a hypothetical cylindrical plane (334) is assumed which contacts the celestial sphere (331) at the equator (303). Then, for a point Q(ψ, δ) on the celestial sphere having a given longitude ψ and a latitude δ, a line segment connecting the center of the celestial sphere and the point Q is extended until it meets the said cylindrical plane. This intersection point is designated as P(ψ, δ). In this manner, the corresponding point P on the cylindrical plane (334) can be obtained for every point Q on the celestial sphere (331) within the said latitude range. Then, a map having a cylindrical projection scheme is obtained when the cylindrical plane is cut and flattened out. Therefore, the lateral coordinate x″ of the point P on the flattened-out cylindrical plane is given by Eq. 5, and the longitudinal coordinate y″ is given by Eq. 6.
Math Figure 5x″=Sψ  [Math.5]Math Figure 6y″=S tan δ  [Math.6]
Such a cylindrical projection scheme is the natural projection scheme for a panoramic camera that produces a panoramic image by rotating in the horizontal plane. Especially, if the lens mounted on the rotating panoramic camera is a distortion-free rectilinear lens, then the resulting panoramic image exactly follows a cylindrical projection scheme. In principle, such a cylindrical projection scheme is the most accurate panoramic projection scheme. However, the panoramic image appears unnatural when the latitude range is large, and thus it is not widely used in practice. Unwrapped panoramic image thus produced and having a cylindrical projection scheme has a lateral width W given by Eq. 3. On the other hand, if the range of the latitude is from δ1 to δ2, then the longitudinal height of the unwrapped panoramic image is given by Eq. 7.
Math Figure 7H=S(tan δ2−tan δ1)  [Math.7]
Therefore, the following equation can be derived from Eq. 3 and Eq. 7.
Math Figure 8
                              W          H                =                              2            ⁢            π                                              tan              ⁢                                                          ⁢                              δ                2                                      -                          tan              ⁢                                                          ⁢                              δ                1                                                                        [                  Math          .                                          ⁢          8                ]            
Therefore, an unwrapped panoramic image following a cylindrical projection scheme must satisfy Eq. 8.
All the animals, plants and inanimate objects such as buildings on the earth are under the influence of gravity, and the direction of gravitational force is the up-right direction or the vertical direction. Ground plane is fairly perpendicular to the gravitational force, but needless to say, it is not so on a slanted ground. Therefore, the word “ground plane” actually refers to the horizontal plane, and the vertical direction is the direction perpendicular to the horizontal plane. Therefore, even if we refer them as the ground plane, the lateral direction, and the longitudinal direction, for the sake of simplicity, the ground plane must be understood as the horizontal plane, the vertical direction must be understood as the direction perpendicular to the horizontal plane, and the horizontal direction must be understood as a direction parallel to the horizontal plane, whenever an exact meaning of a term needs to be clarified.
Panoramic lenses described in references 7 and 8 take panoramic images in one shot with the optical axis of the panoramic lens aligned vertical to the ground plane. A cheaper alternative to the panoramic image acquisition method by the previously described camera with a horizontally-rotating lens consist of taking an image with an ordinary camera with the optical axis horizontally aligned, and repeating to take pictures after horizontally rotating the optical axis by a certain amount. Four to eight pictures are taken in this way, and a panoramic image with a cylindrical projection scheme can be obtained by seamlessly joining the pictures consecutively. Such a technique is called stitching. QuickTime VR from Apple computer inc. is commercial software supporting this stitching technology. This method requires a complex, time-consuming, and elaborate operation of precisely joining several pictures and correcting the lens distortion.
According to the reference 9, another method of obtaining a panoramic or an omnidirectional image is to take a hemispherical image by horizontally pointing a camera equipped with a fisheye lens with more than 180° FOV, and then point the camera to the exact opposite direction and take another hemispherical image. By stitching the two images acquired from the camera using appropriate software, one omnidirectional image having the view of every direction (i.e., 4π steradian) can be obtained. By sending thus obtained image to a geographically separated remote user using communication means such as the Internet, the user can select his own viewpoint from the received omnidirectional image according to his own personal interest, and image processing software on the user's computing device can extract a partial image corresponding to the user-selected viewpoint, and a perspectively correct planar image can be displayed on the computing device. Therefore, through the image processing software, the user can make a choice of turning around (pan), looking-up or down (tilt), or take a close (zoom in) or remote (zoom out) view as if the user is actually at the specific place in the image. This method has a distinctive advantage of multiple users accessing the same Internet site to be able to take looks along the directions of their own choices. This advantage cannot be enjoyed in a panoramic imaging system employing a motion camera such as a pan•tilt camera.
References 10 and 11 describe a method of obtaining an omnidirectional image providing the views of every direction centered on the observer. Despite the lengthy description of the invention, however, the projection scheme provided by the said references is one kind of equidistance projection schemes in essence. In other words, the techniques described in the documents make it possible to obtain omnidirectional images from a real environment or from a cubic panorama, but the obtained omnidirectional image follows an equidistance projection scheme only and its usefulness is thus limited.
On the other hand, reference 12 provides an algorithm for projecting an Omnimax movie on a semi-cylindrical screen using a fisheye lens. Especially, taking into account of the fact that the projection scheme of the fisheye lens mounted on a movie projector deviates from an ideal equidistance projection scheme, a method is described for locating the position of the object point on the film corresponding to a certain point on the screen whereon an image point is formed. Therefore, it is possible to calculate what image has to be on the film in order to project a particular image on the screen, and such an image on the film is produced using a computer. Especially, since the lens distortion is already reflected in the image processing algorithm, a spectator near the movie projector can entertain himself with a satisfactory panoramic image. Nevertheless, the real projection scheme of the fisheye lens in the said reference is inconvenient to use because it has been modeled with the real image height on the film plane as the independent variable, and the zenith angle of the incident ray as the dependent variable. Furthermore, unnecessarily, the real projection scheme of the fisheye lens has been modeled only with an odd polynomial.
Reference 13 provides examples of stereo panoramic images produced by Professor Paul Bourke. Each of the panoramic images follows a cylindrical projection scheme, and a panoramic image of an imaginary scene produced by a computer as well as a panoramic image produced by a rotating slit camera are presented. For panoramic images produced by a computer or produced by a traditional method of rotating slit camera, the lens distortion is not an important issue. However, rotating slit camera cannot be used to take a real-time panoramic image (i.e., movie) of a real world.
References 14 and 15 provide an example of a fisheye lens with 190° FOV, and reference 16 provides various examples of wide-angle lenses including dioptric and catadioptric fisheye lenses with stereographic projection schemes.
On the other hand, reference 17 provides various examples of obtaining panoramic images following cylindrical projection schemes, equi-rectangular projection schemes, and Mercator projection schemes from images acquired using rotationally symmetric wide-angle lenses including fisheye lenses. Referring to FIG. 4 through FIG. 12, most of the examples provided in the said reference can be summarized as follows.
FIG. 4 is a conceptual drawing illustrating the real projection schemes of rotationally symmetric wide-angle lenses (412) including fisheye lenses. Z-axis of the world coordinate system describing objects captured by the wide-angle lens coincides with the optical axis (401) of the wide-angle lens (412). An incident ray (405) having a zenith angle θ with respect to the Z-axis is refracted by the lens (412), and as a refracted ray (406), converges toward an image point P on the focal plane (432). The distance between the nodal point N of the lens and the said focal plane is approximately equal to the effective focal length of the lens. The sub area on the focal plane whereon real image points have been formed is the image plane (433). To obtain a sharp image, the said image plane (433) must coincide with the image sensor plane (413) within the camera body (414). Said focal plane and the said image sensor plane are perpendicular to the optical axis. The intersection point O between the optical axis (401) and the image plane (433) is hereinafter referred to as the first intersection point. The distance between the first intersection point and the said image point P is r.
For general wide-angle lenses, the image height r is given by Eq. 9.
Math Figure 9r=r(θ)  [Math.9]
Here, the unit of the incidence angle θ is radian, and the above function r (θ) is a monotonically increasing function of the zenith angle θ of the incident ray.
Such a real projection scheme of a lens can be experimentally measured using an actual lens, or can be calculated from the lens prescription using dedicated lens design software such as Code V or Zemax. For example, the y-axis coordinate y of the image point on the focal plane by an incident ray having given horizontal and vertical incidence angles can be calculated using a Zemax operator REAY, and the x-axis coordinate x can be similarly calculated using an operator REAX.
FIG. 5 is an imaginary interior scene produced by professor Paul Bourke by using a computer, and it has been assumed that the imaginary lens used to capture the image is a fisheye lens with 180° FOV having an ideal equidistance projection scheme.
This image is a square image, of which both the lateral and the longitudinal dimensions are 250 pixels. Therefore, the coordinate of the optical axis is (125.5, 125.5), and the image height for an incident ray with zenith angle 909s given as r′(π/2)=125.5−1=124.5. Here, r′ is not a physical distance, but an image height measured in pixel distance. Since this imaginary fisheye lens follows an equidistance projection scheme, the projection scheme of this lens is given by Eq. 10.
Math Figure 10
                                          r            ′                    ⁡                      (            θ            )                          =                                            124.5                              (                                  π                  2                                )                                      ⁢            θ                    =                      79.26            ⁢            θ                                              [                  Math          .                                          ⁢          10                ]            
FIG. 6 through FIG. 8 show several embodiments of wide-angle lenses presented in reference 16. FIG. 6 is a dioptric (i.e., refractive) fisheye lens with a stereographic projection scheme, FIG. 7 is a catadioptric fisheye lens with a stereographic projection scheme, and FIG. 8 is a catadioptric panoramic lens with a rectilinear projection scheme. In this manner, the wide-angle lenses from the said reference and in the current invention are not limited to a fisheye lens with an equidistance projection scheme, but encompass all kind of wide-angle lenses that are rotationally symmetric about the optical axes.
The main point of the invention in reference 17 is about methods of obtaining panoramic images by applying mathematically accurate image processing algorithms on images obtained using rotationally symmetric wide-angle lenses. Numerous embodiments in reference 17 can be summarized as follows.
The world coordinate system of the said invention takes the nodal point N of a rotationally symmetric wide-angle lens as the origin, and a vertical line passing through the origin as the Y-axis. Here, the vertical line is a line perpendicular to the ground plane, or more precisely to the horizontal plane (917). The X-axis and the Z-axis of the world coordinate system are contained within the ground plane. The optical axis (901) of the said wide-angle lens generally does not coincide with the Y-axis, and can be contained within the ground plane (i.e., parallel to the ground), or is not contained within the ground plane. The plane (904) containing both the said Y-axis and the said optical axis (901) is referred to as the reference plane. The intersection line (902) between this reference plane (904) and the ground plane (917) coincides with the Z-axis of the world coordinate system. On the other hand, an incident ray (905) originating from an object point Q having a rectangular coordinate (X, Y, Z) in the world coordinate system has an altitude angle δ from the ground plane, and an azimuth angle ψ with respect to the reference plane. The plane (906) containing both the Y-axis and the said incident ray (905) is the incidence plane. The horizontal incidence angle ψ of the said incident ray with respect to the said reference plane is given by Eq. 11.
Math Figure 11
                    ψ        =                              tan                          -              1                                ⁡                      (                          X              Z                        )                                              [                  Math          .                                          ⁢          11                ]            
On the other hand, the vertical incidence angle (i.e., the altitude angle) δ subtended by the said incident ray and the X-Z plane is given by Eq. 12.
Math Figure 12
                    δ        =                              tan                          -              1                                ⁡                      (                          Y                                                                    X                    2                                    +                                      Z                    2                                                                        )                                              [                  Math          .                                          ⁢          12                ]            
The elevation angle μ of the said incident ray is given by Eq. 13, wherein x is an arbitrary angle larger than −90° and smaller than 90°.
Math Figure 13μ=δ−χ  [Math.13]
FIG. 10 is a schematic diagram of a device of the current invention, which also coincides with that of the reference 17, having an imaging system which mainly includes an image acquisition means (1010), an image processing means (1016) and image display means (1015, 1017). The image acquisition means (1010) includes a rotationally symmetric wide-angle lens (1012) and a camera body (1014) having an image sensor (1013) inside. The said wide-angle lens can be a fisheye lens with more than 180° FOV and having an equidistance projection scheme, but it is by no means limited to such a fisheye lens. Rather, it can be any rotationally symmetric wide-angle lens including a catadioptric fisheye lens. Hereinafter, for the sake of notational simplicity, a wide-angle lens is referred to as a fisheye lens. Said camera body contains photoelectronic sensors such as CCD or CMOS sensors, and it can acquire either a still image or a movie. By the said fisheye lens (1012), a real image of the object plane (1031) is formed on the focal plane (1032). In order to obtain a sharp image, the image sensor plane (1013) must coincide with the focal plane (1032).
The real image of the objects on the object plane (1031) formed by the fisheye lens (1012) is converted by the image sensor (1013) into electrical signals, and displayed as an uncorrected image plane (1034) on the image display means (1015). This uncorrected image plane (1034) contains a barrel distortion by the fisheye lens. This distorted image plane can be rectified by the image processing means (1016), and then displayed as a processed image plane (1035) on an image display means (1017) such as a computer monitor or a CCTV monitor. Said image processing can be software image processing by a computer, or hardware image processing by Field Programmable Gate Arrays (FPGA) or ARM core processors.
An arbitrary rotationally symmetric lens including a fisheye lens does not provide said panoramic image or a distortion-free rectilinear image. Therefore, image processing stage is essential in order to obtain a desirable image. FIG. 11 is a conceptual drawing of an uncorrected image plane (1134) prior to the image processing stage, which corresponds to the real image on the image sensor plane (1013). If the lateral dimension of the image sensor plane (1013) is B and the longitudinal dimension is V, then the lateral dimension of the uncorrected image plane is gB and the longitudinal dimension is gV, where g is proportionality constant.
Uncorrected image plane (1134) can be considered as the image displayed on the image display means without rectification of distortion, and is a magnified image of the real image on the image sensor plane by a magnification ratio g. For example, the image sensor plane of a ⅓-inch CCD sensor has a rectangular shape having a lateral dimension of 4.8 mm, and a longitudinal dimension of 3.6 mm. On the other hand, if the monitor is 48 cm in width and 36 cm in height, then the magnification ratio g is 100. More desirably, the side dimension of a pixel in a digital image is considered as 1. A VGA-grade ⅓-inch CCD sensor has pixels in an array form with 640 columns and 480 rows. Therefore, each pixel has a right rectangular shape with both width and height measuring as 4.8 mm/640=7.5 μm, and in this case, the magnification ratio g is given by 1 pixel/7.5 μm=133.3 pixel/mm. In recapitulation, the uncorrected image plane (1134) is a distorted digital image obtained by converting the real image formed on the image sensor plane into electrical signals.
The first intersection point O on the image sensor plane is the intersection point between the optical axis and the image sensor plane. Therefore, a ray entered along the optical axis forms an image point on the said first intersection point O. By definition, the point O′ on the uncorrected image plane corresponding to the first intersection point O in the image sensor plane—hereinafter referred to as the second intersection point—corresponds to the image point by an incident ray entered along the optical axis.
A second rectangular coordinate systems is assumed wherein x′-axis is taken as the axis that passes through the second intersection point O′ on the uncorrected image plane and is parallel to the lateral side of the uncorrected image plane, and y′-axis is taken as the axis that passes through the said second intersection point O′ and is parallel to the longitudinal side of the uncorrected image plane. The positive direction of the x′-axis runs from the left to the right, and the positive direction of the y′-axis runs from the top to the bottom. Then, the lateral coordinate x′ of an arbitrary point on the uncorrected image plane (1134) has a minimum value x′1=gx1 and a maximum value x′2=gx2 (i.e., gx1≦x′≦gx2). In the same manner, the longitudinal coordinate y′ of the said point has a minimum value y′1=gy1 and a maximum value y′2=gy2 (i.e., gy1≦y′≦gy2).
FIG. 12 is a conceptual drawing of a rectified screen of the current invention, wherein the distortion has been removed. In other words, it is a conceptual drawing of a processed image plane (1235) that can be displayed on the image display means. The processed image plane (1235) has a rectangular shape, of which the lateral side measuring as W and the longitudinal side measuring as H. Furthermore, a third rectangular coordinate system is assumed wherein x″-axis is parallel to the lateral side of the processed image plane, and y″-axis is parallel to the longitudinal side of the processed image plane. The z″-axis of the third rectangular coordinate system coincides with the z-axis of the first rectangular coordinate system and the z′-axis of the second rectangular coordinate system. The intersection point O″ between the said z″-axis and the processed image plane—hereinafter referred to as the third intersection point—can take an arbitrary position, and it can even be located outside the processed image plane. Here, the positive direction of the x″-axis runs from the left to the right, and the positive direction of the y″-axis runs from the top to the bottom.
The first and the second intersection points correspond to the location of the optical axis. On the other hand, the third intersection point corresponds not to the location of the optical axis but to the principal direction of vision. The principal direction of vision may coincide with the optical axis, but it is not needed to. Principal direction of vision is the direction of the optical axis of an imaginary panoramic or rectilinear camera corresponding to the desired panoramic or rectilinear images. Hereinafter, for the sake of notational simplicity, the principal direction of vision is referred to as the optical axis direction.
The lateral coordinate x″ of a third point P″ on the processed image plane (1235) has a minimum value x″1 and a maximum value x″2 (i.e., x″1≦x″≦x″2). By definition, the difference between the maximum lateral coordinate and the minimum lateral coordinate is the lateral dimension of the processed image plane (i.e., x″2−x″1=W). In the same manner, the longitudinal coordinate y″ of the third point P″ has a minimum value y″1 and a maximum value y″2 (i.e., y″1≦y″≦y″2). By definition, the difference between the maximum longitudinal coordinate and the minimum longitudinal coordinate is the longitudinal dimension of the processed image plane (i.e., y″2−y″1=H).
The following table 1 summarizes corresponding variables in the object plane, the image sensor plane, the uncorrected image plane, and the processed image plane.
TABLE 1image sensoruncorrectedprocessedsurfaceobject planeplaneimage planeimage planelateral dimension ofBgBWthe planelongitudinalVgVHdimension of theplanecoordinate systemworldthe firstthe secondthe thirdcoordinaterectangularrectangularrectangularsystemcoordinatecoordinatecoordinatesystemsystemsystemlocation of thenodal pointnodal point ofnodal point ofnodal point ofcoordinate originof the lensthe lensthe lensthe lenssymbol of the originOO′O″coordinate axes(X, Y, Z)(x, y, z)(x′, y′, z′)(x″, y″, z″)name of the objectobject pointthe first pointthe second pointthe third pointpoint or the imagepointssymbol of the objectQPP′P″point or the imagepointtwo dimensional(x, y)(x′, y′)(x″, y″)coordinate of theobject point or theimage point
On the other hand, if we assume the coordinate of an image point P″ on the processed image plane (1235) corresponding to an object point with a coordinate (X, Y, Z) in the world coordinate system is (x″, y″), then the said image processing means process the image so that the image point corresponding to an incident ray originating from the said object point appears on the said screen with the coordinate (x″, y″), wherein the lateral coordinate x″ of the image point is given by Eq. 14.
Math Figure 14x″=cψ  [Math.14]
Here, c is proportionality constant.
Furthermore, the longitudinal coordinate y″ of the said image point is given by Eq. 15.
Math Figure 15y″=cF(μ)  [Math.15]
Here, F(μ) is a monotonically increasing function passing through the origin. In mathematical terminology, it means that Eqs. 16 and 17 are satisfied.
Math Figure 16F(0)=0  [Math.16]Math Figure 17
                                          ∂                          F              ⁡                              (                μ                )                                                          ∂            μ                          >        0.                            [                  Math          .                                          ⁢          17                ]            
The above function F can take an arbitrary form, but the most desirable forms are given by Eqs. 18 through 21.
Math Figure 18F(μ)=tan μ  [Math.18]Math Figure 19
                              F          ⁡                      (            μ            )                          =                              tan            ⁢                                                  ⁢            μ                                cos            ⁢                                                  ⁢            χ                                              [                  Math          .                                          ⁢          19                ]            Math Figure 20F(μ)=μ  [Math.20]Math Figure 21
                              F          ⁡                      (            μ            )                          =                  ln          ⁢                      {                          tan              ⁡                              (                                                      μ                    2                                    +                                      π                    4                                                  )                                      }                                              [                  Math          .                                          ⁢          21                ]            
FIG. 13 is a schematic diagram of an ordinary car rear view camera (1310). For a car rear view camera, it is rather common that a wide-angle lens with more than 150° FOV is used, and the optical axis of the lens is typically inclined toward the ground plane (1317) as illustrated in FIG. 13. By installing the camera in this way, parking lane can be easily recognized when backing up the car. Furthermore, since the lens surface is oriented downward toward the ground, precipitation of dust is prevented, and partial protection is provided from rain and snow.
To obtain a panoramic image with a horizontal FOV around 180°, it is desirable to install the image acquisition means (1310) on top of the trunk of a passenger car, and to align the optical axis at a certain angle with the ground plane. Furthermore, a fisheye lens with more than 180° FOV and having an equidistance projection scheme is most preferable, and the image display means is desirably installed next to the driver seat.
It is possible to obtain said panoramic image using a wide-angle camera with its optical axis inclined toward the ground plane. In this case, the world coordinate system takes the nodal point N of the imaging system (1310) as the origin, and takes a vertical line that is perpendicular to the ground plane as the Y-axis, and the Z-axis is set parallel to the car (1351) axle. According to the convention of right handed coordinate system, the positive direction of the X-axis is the direction directly plunging into the paper in FIG. 13. Therefore, if the lens optical axis is inclined below the horizon with an angle α, then a coordinate system fixed to the camera has been rotated around the X-axis of the world coordinate system by angle α. This coordinate system is referred to as the first world coordinate system, and the three axes of this first world coordinate system are named as X′, Y′ and Z′-axis, respectively. In FIG. 13, it appears that the first world coordinate system has been rotated around the X-axis clockwise by angle α relative to the world coordinate system. However, considering the direction of the positive X-axis, it has been in fact rotated counterclockwise by angle α. Since direction of rotation considers counterclockwise rotation as the positive direction, the first world coordinate system in FIG. 13 has been rotated by +α around the X-axis of the world coordinate system.
Regarding the rotation of coordinate system, it is convenient to use the Euler matrices. For this, the coordinate of an object point Q in three dimensional space is designated as a three dimensional vector as given below.
Math Figure 22
                              Q          _                =                              (                                                            X                                                                              Y                                                                              Z                                                      )                    .                                    [                  Math          .                                          ⁢          22                ]            
Here, {right arrow over (Q)} is the three dimensional vector in the world coordinate system starting at the origin and ending at the point Q. Then, the coordinate of a new point obtainable by rotating the point Q in the space by an angle of −α around the X-axis is given by multiplying the matrix given in Eq. 23 on the above vector.
Math Figure 23
                                          M            X                    ⁡                      (            α            )                          =                  (                                                    1                                            0                                            0                                                                    0                                                              cos                  ⁢                                                                          ⁢                  α                                                                              sin                  ⁢                                                                          ⁢                  α                                                                                    0                                                                                  -                    sin                                    ⁢                                                                          ⁢                  α                                                                              cos                  ⁢                                                                          ⁢                  α                                                              )                                    [                  Math          .                                          ⁢          23                ]            
Likewise, the matrix given in Eq. 24 can be used to find the coordinate of a new point which is obtainable by rotating the point Q by angle −β around the Y-axis, and the matrix given in Eq. 25 can be used to find the coordinate of a new point which is obtainable by rotating the point Q by angle −γ around the Z-axis.
Math Figure 24
                                          M            Y                    ⁡                      (            β            )                          =                  (                                                                      cos                  ⁢                                                                          ⁢                  β                                                            0                                                                                  -                    sin                                    ⁢                                                                          ⁢                  β                                                                                    0                                            1                                            0                                                                                      sin                  ⁢                                                                          ⁢                  β                                                            0                                                              cos                  ⁢                                                                          ⁢                  β                                                              )                                    [                  Math          .                                          ⁢          24                ]            Math Figure 25
                                          M            Z                    ⁡                      (            γ            )                          =                  (                                                                      cos                  ⁢                                                                          ⁢                  γ                                                                              sin                  ⁢                                                                          ⁢                  γ                                                            0                                                                                                          -                    sin                                    ⁢                                                                          ⁢                  γ                                                                              cos                  ⁢                                                                          ⁢                  γ                                                            0                                                                    0                                            0                                            1                                              )                                    [                  Math          .                                          ⁢          25                ]            
Matrices in Eqs. 23 through 25 can describe the case where the coordinate system is fixed and the point in space has been rotated, but also the same matrices can describe the case where the point in space is fixed and the coordinate system has been rotated in the reverse direction. These two cases are mathematically equivalent. Therefore, the coordinate of a point Q in the first world coordinate system which is obtained by rotating the world coordinate system by angle α around the X-axis as indicated in FIG. 13 is given by Eq. 26.
Math Figure 26
                                          Q            _                    ′                =                              (                                                                                X                    ′                                                                                                                    Y                    ′                                                                                                                    Z                    ′                                                                        )                    =                                                    M                X                            ⁡                              (                α                )                                      ⁢                          (                                                                    X                                                                                        Y                                                                                        Z                                                              )                                                          [                  Math          .                                          ⁢          26                ]            
Using the matrix given in Eq. 23, the coordinate in the first world coordinate system can be given as follows in terms of the coordinate in the world coordinate system.
Math Figure 27X′=X  [Math.27]Math Figure 28Y′=Y cos α+Z sin α  [Math.28]Math Figure 29Z′=−Y sin α+Z cos α  [Math.29]
Referring to FIG. 13, let's assume that an imaging system has been installed with its optical axis inclined toward the ground plane, and nevertheless, it is desired to obtain a panoramic image that is parallel to the ground plane. If we assume that FIG. 5 is a wide-angle image with α=0°, then FIG. 14 is a wide-angle image with α=30°. In this case, the following algorithm can be used to obtain a panoramic image that is parallel to the ground plane. First, under the assumption that the said imaging system is parallel to the ground plane, the size (W, H) of the processed image plane, the location of the third intersection point O″, and the horizontal field of view Δψ are determined. For simplicity of argument, the horizontal FOV is assumed as symmetric. Then, the horizontal incidence angle ψ corresponding to the lateral coordinate x″ on the processed image plane and the vertical incidence angle δ corresponding to the longitudinal coordinate y″ are given by Eqs. 30 through 32.
Math Figure 30
                    A        =                  W          Δψ                                    [                  Math          .                                          ⁢          30                ]            Math Figure 31
                    ψ        =                              x            ″                    A                                    [                  Math          .                                          ⁢          31                ]            Math Figure 32
                    δ        =                              F                          -              1                                ⁡                      (                                          y                ″                            A                        )                                              [                  Math          .                                          ⁢          32                ]            
Here, the angle χ is 0°. Therefore, the function F(δ) is identical to the function F(μ) given in Eqs. 16 through 21. In other words, it is true that F(δ)=F(μ). Next, it is assumed that an incident ray having these horizontal and vertical incidence angles has been originated from an object point on a hemisphere with a radius S having its center at the nodal point of the lens. Then, the coordinate of the said object point in the world coordinate system is given by Eqs. 33 through 36.
Math Figure 33X=S cos δ sin ψ  [Math.33]Math Figure 34Y=S sin δ  [Math.34]Math Figure 35Z=S cos δ cos ψ  [Math.35]Math Figure 36S=√{square root over (X2+Y2+Z2)}
The coordinate of this object point in the first world coordinate system is given by Eqs. 27 through 29. The X′, Y′ and Z′-axes of this first world coordinate system are parallel to the x, y, and z-axes of the first rectangular coordinate system, respectively. Therefore, the zenith and the azimuth angles of the incident ray are given by Eqs. 37 and 38.
Math Figure 37
                    θ        =                              cos                          -              1                                ⁡                      (                                          Z                ′                            S                        )                                              [                  Math          .                                          ⁢          37                ]            Math Figure 38
                    ϕ        =                              tan                          -              1                                ⁡                      (                                          Y                ′                                            X                ′                                      )                                              [                  Math          .                                          ⁢          38                ]            
Finally, two dimensional rectangular coordinate (x′, y′) of the second point in the uncorrected image plane having these zenith and azimuth angles can be obtained from the two dimensional polar coordinate by Eqs. 39 and 40.
Math Figure 39x′=gr(θ)cos φ  [Math.39]Math Figure 40y′=gr(θ)sin φ  [Math.40]
Using Eqs. 9 through 40, a panoramic image having an ideal projection scheme can be extracted from an image acquired using a fisheye lens exhibiting the distortion aberration. First, depending on the user's need, a desirable size (W, H) of the panoramic image, and the location of the third intersection point O″ are determined. The said third intersection point can be located even outside the processed image plane. In other words, the range of the lateral coordinate (x″1≦x″≦x″2) on the processed image plane as well as the range of the longitudinal coordinate (y″1≦y″<y″2) can take arbitrary real numbers. Also, the horizontal FOV Δψ of this panoramic image (i.e., the processed image plane) is determined. The functional form of F(δ) dictating the desirable projection scheme along the vertical direction is determined, as well. Then, the horizontal incidence angle w and the vertical incidence angle δ of an incident ray corresponding to the third point on the panoramic image having a rectangular coordinate (x″, y″) can be obtained using Eqs. 30 through 32. Then, the zenith angle θ and the azimuth angle φ of an incident ray having the said horizontal incidence angle and the vertical incidence angle are calculated using Eqs. 37 and 38. Next, the real image height r corresponding to the zenith angle θ of the incident ray is obtained using Eq. 9. Utilizing the real image height r, the magnification ratio g, and the azimuth angle φ of the incident ray, the rectangular coordinate (x′, y′) of the image point on the uncorrected image plane is obtained using Eqs. 39 and 40. In this procedure, the coordinate of the second intersection point on the uncorrected image plane, or equivalently the first intersection point on the image sensor plane, has to be accurately determined. Such a location of the intersection point can be easily found using various methods including image processing method. Since such technique is well known to the people in this field, it will not be described in this document. Finally, the video signal (i.e., RGB signal) by the fisheye lens from the image point having said rectangular coordinate is given as the video signal for the image point on the panoramic image having the rectangular coordinate (x″, y″). A panoramic image having an ideal projection scheme can be obtained by image processing for all the image points on the processed image plane by the above-described method.
Considering the fact that all the image sensors and display devices are digital devices, image processing procedure must use the following set of equations. First, a desirable size of the processed image plane and the location (Io, Jo) of the third intersection point are determined. Here, the location (Io, Jo) of the third intersection point refers to the pixel coordinate of the third intersection point O″. Conventionally, the coordinate of a pixel on the upper left corner of a digitized image is defined as (1, 1) or (0, 0). In the current invention, we will assume that the coordinate of the pixel on the upper left corner is given as (1, 1). Then, the horizontal incidence angle ψ1 corresponding to the lateral pixel coordinate of J=1 and the horizontal incidence angle ψJmax corresponding to the lateral pixel coordinate of J=Jmax are determined. Then, for every pixel (I, J) on the processed image plane, the horizontal incidence angle ψJ and the vertical incidence angle δ1 are calculated using Eqs. 41 and 42.
Math Figure 41
                              ψ          J                =                                                            ψ                Jmax                            -                              ψ                1                                                                    J                max                            -              1                                ⁢                      (                          J              -                              J                o                                      )                                              [                  Math          .                                          ⁢          41                ]            Math Figure 42
                              δ          I                =                              F                          -              1                                ⁢                      {                                                                                ψ                    Jmax                                    -                                      ψ                    1                                                                                        J                    max                                    -                  1                                            ⁢                              (                                  I                  -                                      I                    o                                                  )                                      }                                              [                  Math          .                                          ⁢          42                ]            
From the above horizontal and vertical incidence angles, the coordinate of an imaginary object point in the world coordinate system is calculated using Eqs. 43 through 45.
Math Figure 43XI,J=cos δI sin ψJ  [Math.43]Math Figure 44YI,J=sin δI  [Math.44]Math Figure 45ZI,J=cos δI cos ψJ  [Math.45]
Here, the object distance S does not affect the final outcome and thus it is assumed as 1 for the sake of simplicity. From this coordinate of the object point in the world coordinate system, the coordinate of the object point in the first world coordinate system is calculated from Eqs. 46 through 48.
Math Figure 46X′I,J=XI,J  [Math.46]Math Figure 47Y′I,J=YI,J cos α+ZI,J sin α  [Math.47]Math Figure 48Z′I,J=−YI,J sin α+ZI,J cos α  [Math.48]
From this coordinate, the zenith angle θI,J and the azimuth angle ΦI,J of the incident ray are calculated using Eqs. 49 and 50,
Math Figure 49θI,J=cos−1(Z′I,J)  [Math.49]Math Figure 50
                              ϕ                      I            ,            J                          =                              tan                          -              1                                ⁡                      (                                          Y                                  I                  ,                  J                                ′                                            X                                  I                  ,                  J                                ′                                      )                                              [                  Math          .                                          ⁢          50                ]            
Next, the image height rI,J on the image sensor plane is calculated using Eq. 51.
Math Figure 51rI,J=r(θI,J)  [Math.51]
Then, the position (Ko, Lo) of the second intersection point on the uncorrected image plane and the magnification ratio g are used to find the position of the second point on the uncorrected image plane.
Math Figure 52x′I,J=Lo+grI,J cos(φI,J)  [Math.52]Math Figure 53y′I,J=Ko+grI,J sin(φI,J)  [Math.53]
Once the position of the corresponding second point has been found, diverse interpolation methods such as the nearest-neighbor method, the bilinear interpolation method, and the bicubic interpolation method can be used to obtain a panoramic image.
FIG. 15 is a panoramic image obtained using this method, and a cylindrical projection scheme has been employed. As can be seen from FIG. 15, a perfect panoramic image has been obtained despite the fact that the optical axis is not parallel to the ground plane. Using such a panoramic imaging system as a car rear view camera, the backside of a vehicle can be entirely monitored without any dead spot.
One point which needs special attention when using such an imaging system as a car rear view camera is the fact that for a device (i.e., a car) of which the moving direction is the exact opposite of the optical axis direction of the image acquisition means, it can cause a great confusion to the driver if a panoramic image obtained by above described method is displayed without any further processing. Since the car rear view camera is heading toward the backside of the car, the right end of the car appears as the left end on the monitor showing the images captured by the rear view camera. However, the driver can fool himself by thinking that the image is showing the left end of the car from his own viewpoint of looking at the front end of the car, and thus, there is a great danger of possible accidents. To prevent such a perilous confusion, it is important to switch the left and the right sides of the image obtained using a car rear view camera before displaying it on the monitor. The video signal S′(I, J) for the pixel in a mirrored (i.e., the left and the right sides are exchanged) processed image plane with a coordinate (I, J) is given by the video signal S(I, Jmax−J+1) from the pixel in the processed image plane with coordinate (I, Jmax−J+1).
Math Figure 54S′(I,J)=S(I,Jmax−J+1)  [Math.54]
Identical image acquisition means can be installed near the room mirror, frontal bumper, or the radiator grill in order to be used as a recording camera connected to a car black box for the purpose of recording vehicle's driving history.
Above embodiment has been described in relation to a car rear view camera, but it must be obvious that the usefulness of the invention described in this embodiment is not limited to a car rear view camera.
Said invention in reference 17 provides image processing algorithms for extracting mathematically accurate panoramic images and devices implementing the algorithms. In many cases, however, distortion-free rectilinear image can be of more value. Or, it can be more satisfactory when panoramic images and rectilinear images are both available.    [reference 1] J. F. Blinn and M. E. Newell, “Texture and reflection in computer generated images”, Communications of the ACM, 19, 542-547 (1976).    [reference 2] N. Greene, “Environment mapping and other applications of world projections”, IEEE Computer Graphics and Applications, 6, 21-29 (1986).    [reference 3] S. D. Zimmermann, “Omniview motionless camera orientation system”, U.S. Pat. No. 5,185,667, date of patent Feb. 9, 1993.    [reference 4] E. Gullichsen and S. Wyshynski, “Wide-angle image dewarping method and apparatus”, U.S. Pat. No. 6,005,611, date of patent Dec. 21, 1999.    [reference 5] E. W. Weisstein, “Cylindrical Projection”, http://mathworld.wolfram.com/CylindricalProjection.html.    [reference 6] W. D. G. Cox, “An introduction to the theory of perspective—part 1”, The British Journal of Photography, 4, 628-634 (1969).    [reference 7] G. Kweon, K. Kim, Y. Choi, G. Kim, and S. Yang, “Catadioptric panoramic lens with a rectilinear projection scheme”, Journal of the Korean Physical Society, 48, 554-563 (2006).    [reference 8] G. Kweon, Y. Choi, G. Kim, and S. Yang, “Extraction of perspectively normal images from video sequences obtained using a catadioptric panoramic lens with the rectilinear projection scheme”, Technical Proceedings of the 10th World Multi-Conference on Systemics, Cybernetics, and Informatics, 67-75 (Orlando, Fla., USA, June, 2006).    [reference 9] H. L. Martin and D. P. Kuban, “System for omnidirectional image viewing at a remote location without the transmission of control signals to select viewing parameters”, U.S. Pat. No. 5,384,588, date of patent Jan. 24, 1995.    [reference 10] F. Oxaal, “Method and apparatus for performing perspective transformation on visible stimuli”, U.S. Pat. No. 5,684,937, date of patent Nov. 4, 1997.    [reference 11] F. Oxaal, “Method for generating and interactively viewing spherical image data”, U.S. Pat. No. 6,271,853, date of patent Aug. 7, 2001.    [reference 12] N. L. Max, “Computer graphics distortion for IMAX and OMNIMAX projection”, Proc. NICOGRAPH, 137-159 (1983).    [reference 13] P. D. Bourke, “Synthetic stereoscopic panoramic images”, Lecture Notes in Computer Graphics (LNCS), Springer, 4270, 147-155 (2006).    [reference 14] G. Kweon and M. Laikin, “Fisheye lens”, Korean patent 10-0888922, date of patent Mar. 10, 2009.    [reference 15] G. Kweon, Y. Choi, and M. Laikin, “Fisheye lens for image processing applications”, J. of the Optical Society of Korea, 12, 79-87 (2008).    [reference 16] G. Kweon and M. Laikin, “Wide-angle lenses”, Korean patent 10-0826571, date of patent Apr. 24, 2008.    [reference 17] G. Kweon, “Methods of obtaining panoramic images using rotationally symmetric wide-angle lenses and devices thereof”, Korean patent 10-0882011, date of patent Jan. 29, 2009.    [reference 18] W. K. Pratt, Digital Image Processing, 3rd edition (John Wiley, New York, 2001), Chap. 19.